{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AutoML with Tabular data - Data Processing\n",
    "\n",
    "The complete AutoML pipeline that translates raw data into accurate predictions involves many stages abstracted by AutoGluon's one-line `fit()`, such as:\n",
    "- Data splitting\n",
    "- Data preprocessing\n",
    "- Training of individual models\n",
    "- Hyperparameter-tuning (optional)\n",
    "- Model ensembling (optional)\n",
    "- Feature-engineering/selection (optional)\n",
    "\n",
    "Here we describe some basic principles to improve various stages of the pipeline, starting with how to process the given dataset. This tutorial focuses on subtle yet practically-important issues, assuming you're already familiar with overall forms of standard data processing required in most ML projects. If not, please first look at these resources: ([Rencberoglu, 2019](https://towardsdatascience.com/feature-engineering-for-machine-learning-3a5e293a5114); [D'yakonov & Semenov, 2019](https://www.coursera.org/lecture/competitive-data-science/overview-1Nh5Q)).\n",
    "\n",
    "\n",
    "## Data Splitting\n",
    "\n",
    "**How much data to hold-out**:  Predictive performance on validation data crucially guides automated decisions about which model (or combination of models) is best, how many training iterations (i.e. epochs/boosting-rounds) to apply for iteratively-optimized models, what hyperparameter-values to use, etc. It's thus critical to use a representative validation set that facilitates accurate estimation of generalization performance. However, we do not want to hold-out too much data for validation, since then less data is available for actually training the models. Once the validation-set reaches a certain size that we can accurately estimate predictive performance, additional validation-set increases will only marginally improve our estimates. While the size of the validation data determines the *variance* of our performance-estimates, how *biased* these estimates are is determined by the number of modeling decisions we base on validation-performance. \n",
    "By default, AutoGluon selects the fraction of data to hold-out for validation as follows:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "if num_train_rows < 5000:\n",
    "    holdout_frac = max(0.1, min(0.2, 500.0 / num_train_rows))\n",
    "else:\n",
    "    holdout_frac = max(0.01, min(0.1, 2500.0 / num_train_rows))\n",
    "\n",
    "if hyperparameter_tune:\n",
    "    holdout_frac = min(0.2, holdout_frac * 2)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Between 5,000-25,000 examples, we hold-out 10% of the data, as we want to grow validation set to a stable 2500 examples, but for larger sample-sizes, we only hold-out 1% of the data which suffices to accurately estimate validation performance. If hyperparameter-tuning is performed, we double the size of the validation set to mitigate the bias introduced from choosing many hyperparameter-values based on the same validation data.\n",
    "\n",
    "For smaller datasets, it can be desirable to utilize multiple train/validation splits, i.e. via [k-fold cross-validation](https://www.datavedas.com/k-fold-cross-validation/), which we discuss in the next Notebook. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Class-stratification:** In classification problems, we make sure to stratify labels between the training and validation data. Stratification simply means the proportions of each class are matched between training and validation data. This prevents a shift in the class-label-distribution that might otherwise arise simply due to random chance.  Extremely rare classes remain an issue, and AutoGluon by default simply discards all data from classes that occur <10 times (can be adjusted via `label_count_threshold` [argument of `fit()`](https://autogluon.mxnet.io/api/autogluon.task.html#autogluon.task.TabularPrediction.fit)). AutoGluon-trained models thus never predict these discarded classes (about which models could anyway learn very little), but AutoGluon automatically remembers to take them into account when evaluating predictive performance.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Manually specify validation data:** Whether a random training/validation split is appropriate for a particular application is hard for an AutoML system to determine. Thus AutoGluon places the burden on the user to decide this. If you have reason to believe your future test data will stem from a different distribution than the training data, you should aim to provide the validation set you believe is most representative of the test distribution. For example: data are often collected at different times, where the underlying predictive relationships vary temporally, and the goal is predict on data received at future times. Here a reasonable strategy may be to reserve your most recent data as the validation-set, to mimic the time-difference from the training data that will be encountered during inference. With AutoGluon, we can easily specify a validation-set via the `tuning_data` argument as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loaded data from: https://autogluon.s3.amazonaws.com/datasets/diabetes/train.csv | Columns = 47 / 47 | Rows = 61059 -> 61059\n",
      "Loaded data from: https://autogluon.s3.amazonaws.com/datasets/diabetes/validation.csv | Columns = 47 / 47 | Rows = 20353 -> 20353\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gender</th>\n",
       "      <th>age</th>\n",
       "      <th>weight</th>\n",
       "      <th>admission_type_id</th>\n",
       "      <th>discharge_disposition_id</th>\n",
       "      <th>admission_source_id</th>\n",
       "      <th>time_in_hospital</th>\n",
       "      <th>payer_code</th>\n",
       "      <th>medical_specialty</th>\n",
       "      <th>num_lab_procedures</th>\n",
       "      <th>...</th>\n",
       "      <th>citoglipton</th>\n",
       "      <th>insulin</th>\n",
       "      <th>glyburide-metformin</th>\n",
       "      <th>glipizide-metformin</th>\n",
       "      <th>glimepiride-pioglitazone</th>\n",
       "      <th>metformin-rosiglitazone</th>\n",
       "      <th>metformin-pioglitazone</th>\n",
       "      <th>change</th>\n",
       "      <th>diabetesMed</th>\n",
       "      <th>readmitted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Female</td>\n",
       "      <td>[0-10)</td>\n",
       "      <td>?</td>\n",
       "      <td>\"6\"</td>\n",
       "      <td>\"25\"</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>1.5</td>\n",
       "      <td>?</td>\n",
       "      <td>Pediatrics-Endocrinology</td>\n",
       "      <td>41</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>NO</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Female</td>\n",
       "      <td>[10-20)</td>\n",
       "      <td>?</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"7\"</td>\n",
       "      <td>3.0</td>\n",
       "      <td>?</td>\n",
       "      <td>?</td>\n",
       "      <td>59</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>Up</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Ch</td>\n",
       "      <td>Yes</td>\n",
       "      <td>&gt;30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Female</td>\n",
       "      <td>[20-30)</td>\n",
       "      <td>?</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"7\"</td>\n",
       "      <td>2.3</td>\n",
       "      <td>?</td>\n",
       "      <td>?</td>\n",
       "      <td>11</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>NO</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Male</td>\n",
       "      <td>[30-40)</td>\n",
       "      <td>?</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"7\"</td>\n",
       "      <td>2.3</td>\n",
       "      <td>?</td>\n",
       "      <td>?</td>\n",
       "      <td>44</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>Up</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Ch</td>\n",
       "      <td>Yes</td>\n",
       "      <td>NO</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Male</td>\n",
       "      <td>[40-50)</td>\n",
       "      <td>?</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"7\"</td>\n",
       "      <td>1.7</td>\n",
       "      <td>?</td>\n",
       "      <td>?</td>\n",
       "      <td>51</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>Steady</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Ch</td>\n",
       "      <td>Yes</td>\n",
       "      <td>NO</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>595</th>\n",
       "      <td>Male</td>\n",
       "      <td>[40-50)</td>\n",
       "      <td>?</td>\n",
       "      <td>\"6\"</td>\n",
       "      <td>\"25\"</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>3.0</td>\n",
       "      <td>?</td>\n",
       "      <td>InternalMedicine</td>\n",
       "      <td>41</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>NO</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>596</th>\n",
       "      <td>Male</td>\n",
       "      <td>[70-80)</td>\n",
       "      <td>?</td>\n",
       "      <td>\"2\"</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"20\"</td>\n",
       "      <td>3.2</td>\n",
       "      <td>?</td>\n",
       "      <td>?</td>\n",
       "      <td>11</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>&gt;30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>597</th>\n",
       "      <td>Female</td>\n",
       "      <td>[30-40)</td>\n",
       "      <td>?</td>\n",
       "      <td>\"6\"</td>\n",
       "      <td>\"25\"</td>\n",
       "      <td>\"7\"</td>\n",
       "      <td>4.3</td>\n",
       "      <td>?</td>\n",
       "      <td>Psychiatry</td>\n",
       "      <td>30</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>Up</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Ch</td>\n",
       "      <td>Yes</td>\n",
       "      <td>NO</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>598</th>\n",
       "      <td>Female</td>\n",
       "      <td>[90-100)</td>\n",
       "      <td>?</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"6\"</td>\n",
       "      <td>\"5\"</td>\n",
       "      <td>7.7</td>\n",
       "      <td>?</td>\n",
       "      <td>InternalMedicine</td>\n",
       "      <td>34</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>NO</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>599</th>\n",
       "      <td>Male</td>\n",
       "      <td>[50-60)</td>\n",
       "      <td>?</td>\n",
       "      <td>\"2\"</td>\n",
       "      <td>\"6\"</td>\n",
       "      <td>\"20\"</td>\n",
       "      <td>4.9</td>\n",
       "      <td>?</td>\n",
       "      <td>?</td>\n",
       "      <td>36</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>Steady</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Ch</td>\n",
       "      <td>Yes</td>\n",
       "      <td>&gt;30</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>600 rows × 47 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     gender       age weight admission_type_id discharge_disposition_id  \\\n",
       "0    Female    [0-10)      ?               \"6\"                     \"25\"   \n",
       "1    Female   [10-20)      ?               \"1\"                      \"1\"   \n",
       "2    Female   [20-30)      ?               \"1\"                      \"1\"   \n",
       "3      Male   [30-40)      ?               \"1\"                      \"1\"   \n",
       "4      Male   [40-50)      ?               \"1\"                      \"1\"   \n",
       "..      ...       ...    ...               ...                      ...   \n",
       "595    Male   [40-50)      ?               \"6\"                     \"25\"   \n",
       "596    Male   [70-80)      ?               \"2\"                      \"1\"   \n",
       "597  Female   [30-40)      ?               \"6\"                     \"25\"   \n",
       "598  Female  [90-100)      ?               \"1\"                      \"6\"   \n",
       "599    Male   [50-60)      ?               \"2\"                      \"6\"   \n",
       "\n",
       "    admission_source_id  time_in_hospital payer_code  \\\n",
       "0                   \"1\"               1.5          ?   \n",
       "1                   \"7\"               3.0          ?   \n",
       "2                   \"7\"               2.3          ?   \n",
       "3                   \"7\"               2.3          ?   \n",
       "4                   \"7\"               1.7          ?   \n",
       "..                  ...               ...        ...   \n",
       "595                 \"1\"               3.0          ?   \n",
       "596                \"20\"               3.2          ?   \n",
       "597                 \"7\"               4.3          ?   \n",
       "598                 \"5\"               7.7          ?   \n",
       "599                \"20\"               4.9          ?   \n",
       "\n",
       "            medical_specialty  num_lab_procedures  ...  citoglipton  insulin  \\\n",
       "0    Pediatrics-Endocrinology                  41  ...           No       No   \n",
       "1                           ?                  59  ...           No       Up   \n",
       "2                           ?                  11  ...           No       No   \n",
       "3                           ?                  44  ...           No       Up   \n",
       "4                           ?                  51  ...           No   Steady   \n",
       "..                        ...                 ...  ...          ...      ...   \n",
       "595          InternalMedicine                  41  ...           No       No   \n",
       "596                         ?                  11  ...           No       No   \n",
       "597                Psychiatry                  30  ...           No       Up   \n",
       "598          InternalMedicine                  34  ...           No       No   \n",
       "599                         ?                  36  ...           No   Steady   \n",
       "\n",
       "     glyburide-metformin  glipizide-metformin  glimepiride-pioglitazone  \\\n",
       "0                     No                   No                        No   \n",
       "1                     No                   No                        No   \n",
       "2                     No                   No                        No   \n",
       "3                     No                   No                        No   \n",
       "4                     No                   No                        No   \n",
       "..                   ...                  ...                       ...   \n",
       "595                   No                   No                        No   \n",
       "596                   No                   No                        No   \n",
       "597                   No                   No                        No   \n",
       "598                   No                   No                        No   \n",
       "599                   No                   No                        No   \n",
       "\n",
       "    metformin-rosiglitazone metformin-pioglitazone change  diabetesMed  \\\n",
       "0                        No                     No     No           No   \n",
       "1                        No                     No     Ch          Yes   \n",
       "2                        No                     No     No          Yes   \n",
       "3                        No                     No     Ch          Yes   \n",
       "4                        No                     No     Ch          Yes   \n",
       "..                      ...                    ...    ...          ...   \n",
       "595                      No                     No     No           No   \n",
       "596                      No                     No     No           No   \n",
       "597                      No                     No     Ch          Yes   \n",
       "598                      No                     No     No           No   \n",
       "599                      No                     No     Ch          Yes   \n",
       "\n",
       "    readmitted  \n",
       "0           NO  \n",
       "1          >30  \n",
       "2           NO  \n",
       "3           NO  \n",
       "4           NO  \n",
       "..         ...  \n",
       "595         NO  \n",
       "596        >30  \n",
       "597         NO  \n",
       "598         NO  \n",
       "599        >30  \n",
       "\n",
       "[600 rows x 47 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gender</th>\n",
       "      <th>age</th>\n",
       "      <th>weight</th>\n",
       "      <th>admission_type_id</th>\n",
       "      <th>discharge_disposition_id</th>\n",
       "      <th>admission_source_id</th>\n",
       "      <th>time_in_hospital</th>\n",
       "      <th>payer_code</th>\n",
       "      <th>medical_specialty</th>\n",
       "      <th>num_lab_procedures</th>\n",
       "      <th>...</th>\n",
       "      <th>citoglipton</th>\n",
       "      <th>insulin</th>\n",
       "      <th>glyburide-metformin</th>\n",
       "      <th>glipizide-metformin</th>\n",
       "      <th>glimepiride-pioglitazone</th>\n",
       "      <th>metformin-rosiglitazone</th>\n",
       "      <th>metformin-pioglitazone</th>\n",
       "      <th>change</th>\n",
       "      <th>diabetesMed</th>\n",
       "      <th>readmitted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Male</td>\n",
       "      <td>[60-70)</td>\n",
       "      <td>?</td>\n",
       "      <td>\"2\"</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"4\"</td>\n",
       "      <td>12.0</td>\n",
       "      <td>MD</td>\n",
       "      <td>Radiologist</td>\n",
       "      <td>68</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>Steady</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Ch</td>\n",
       "      <td>Yes</td>\n",
       "      <td>NO</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Female</td>\n",
       "      <td>[60-70)</td>\n",
       "      <td>?</td>\n",
       "      <td>\"2\"</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"7\"</td>\n",
       "      <td>2.2</td>\n",
       "      <td>MC</td>\n",
       "      <td>Emergency/Trauma</td>\n",
       "      <td>36</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>Steady</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Ch</td>\n",
       "      <td>Yes</td>\n",
       "      <td>&gt;30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Female</td>\n",
       "      <td>[70-80)</td>\n",
       "      <td>?</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"7\"</td>\n",
       "      <td>3.2</td>\n",
       "      <td>MC</td>\n",
       "      <td>?</td>\n",
       "      <td>59</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>NO</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Male</td>\n",
       "      <td>[70-80)</td>\n",
       "      <td>?</td>\n",
       "      <td>\"2\"</td>\n",
       "      <td>\"2\"</td>\n",
       "      <td>\"7\"</td>\n",
       "      <td>5.6</td>\n",
       "      <td>SP</td>\n",
       "      <td>Emergency/Trauma</td>\n",
       "      <td>61</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>Steady</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Ch</td>\n",
       "      <td>Yes</td>\n",
       "      <td>NO</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Female</td>\n",
       "      <td>[80-90)</td>\n",
       "      <td>?</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"3\"</td>\n",
       "      <td>\"7\"</td>\n",
       "      <td>12.9</td>\n",
       "      <td>?</td>\n",
       "      <td>InternalMedicine</td>\n",
       "      <td>73</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>Up</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Ch</td>\n",
       "      <td>Yes</td>\n",
       "      <td>&lt;30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>595</th>\n",
       "      <td>Female</td>\n",
       "      <td>[60-70)</td>\n",
       "      <td>?</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"3\"</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>11.6</td>\n",
       "      <td>MC</td>\n",
       "      <td>?</td>\n",
       "      <td>81</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>Up</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Ch</td>\n",
       "      <td>Yes</td>\n",
       "      <td>&gt;30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>596</th>\n",
       "      <td>Female</td>\n",
       "      <td>[80-90)</td>\n",
       "      <td>?</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"3\"</td>\n",
       "      <td>\"7\"</td>\n",
       "      <td>6.6</td>\n",
       "      <td>MC</td>\n",
       "      <td>Emergency/Trauma</td>\n",
       "      <td>65</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>&gt;30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>597</th>\n",
       "      <td>Male</td>\n",
       "      <td>[80-90)</td>\n",
       "      <td>?</td>\n",
       "      <td>\"2\"</td>\n",
       "      <td>\"5\"</td>\n",
       "      <td>\"7\"</td>\n",
       "      <td>6.3</td>\n",
       "      <td>MD</td>\n",
       "      <td>Emergency/Trauma</td>\n",
       "      <td>45</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>Down</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Ch</td>\n",
       "      <td>Yes</td>\n",
       "      <td>NO</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>598</th>\n",
       "      <td>Female</td>\n",
       "      <td>[70-80)</td>\n",
       "      <td>?</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"7\"</td>\n",
       "      <td>5.3</td>\n",
       "      <td>?</td>\n",
       "      <td>InternalMedicine</td>\n",
       "      <td>53</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>NO</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>599</th>\n",
       "      <td>Female</td>\n",
       "      <td>[60-70)</td>\n",
       "      <td>?</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"7\"</td>\n",
       "      <td>4.7</td>\n",
       "      <td>MC</td>\n",
       "      <td>?</td>\n",
       "      <td>32</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>NO</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>600 rows × 47 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     gender      age weight admission_type_id discharge_disposition_id  \\\n",
       "0      Male  [60-70)      ?               \"2\"                      \"1\"   \n",
       "1    Female  [60-70)      ?               \"2\"                      \"1\"   \n",
       "2    Female  [70-80)      ?               \"1\"                      \"1\"   \n",
       "3      Male  [70-80)      ?               \"2\"                      \"2\"   \n",
       "4    Female  [80-90)      ?               \"1\"                      \"3\"   \n",
       "..      ...      ...    ...               ...                      ...   \n",
       "595  Female  [60-70)      ?               \"1\"                      \"3\"   \n",
       "596  Female  [80-90)      ?               \"1\"                      \"3\"   \n",
       "597    Male  [80-90)      ?               \"2\"                      \"5\"   \n",
       "598  Female  [70-80)      ?               \"1\"                      \"1\"   \n",
       "599  Female  [60-70)      ?               \"1\"                      \"1\"   \n",
       "\n",
       "    admission_source_id  time_in_hospital payer_code medical_specialty  \\\n",
       "0                   \"4\"              12.0         MD       Radiologist   \n",
       "1                   \"7\"               2.2         MC  Emergency/Trauma   \n",
       "2                   \"7\"               3.2         MC                 ?   \n",
       "3                   \"7\"               5.6         SP  Emergency/Trauma   \n",
       "4                   \"7\"              12.9          ?  InternalMedicine   \n",
       "..                  ...               ...        ...               ...   \n",
       "595                 \"1\"              11.6         MC                 ?   \n",
       "596                 \"7\"               6.6         MC  Emergency/Trauma   \n",
       "597                 \"7\"               6.3         MD  Emergency/Trauma   \n",
       "598                 \"7\"               5.3          ?  InternalMedicine   \n",
       "599                 \"7\"               4.7         MC                 ?   \n",
       "\n",
       "     num_lab_procedures  ...  citoglipton  insulin  glyburide-metformin  \\\n",
       "0                    68  ...           No   Steady                   No   \n",
       "1                    36  ...           No   Steady                   No   \n",
       "2                    59  ...           No       No                   No   \n",
       "3                    61  ...           No   Steady                   No   \n",
       "4                    73  ...           No       Up                   No   \n",
       "..                  ...  ...          ...      ...                  ...   \n",
       "595                  81  ...           No       Up                   No   \n",
       "596                  65  ...           No       No                   No   \n",
       "597                  45  ...           No     Down                   No   \n",
       "598                  53  ...           No       No                   No   \n",
       "599                  32  ...           No       No                   No   \n",
       "\n",
       "     glipizide-metformin  glimepiride-pioglitazone metformin-rosiglitazone  \\\n",
       "0                     No                        No                      No   \n",
       "1                     No                        No                      No   \n",
       "2                     No                        No                      No   \n",
       "3                     No                        No                      No   \n",
       "4                     No                        No                      No   \n",
       "..                   ...                       ...                     ...   \n",
       "595                   No                        No                      No   \n",
       "596                   No                        No                      No   \n",
       "597                   No                        No                      No   \n",
       "598                   No                        No                      No   \n",
       "599                   No                        No                      No   \n",
       "\n",
       "    metformin-pioglitazone change  diabetesMed readmitted  \n",
       "0                       No     Ch          Yes         NO  \n",
       "1                       No     Ch          Yes        >30  \n",
       "2                       No     No           No         NO  \n",
       "3                       No     Ch          Yes         NO  \n",
       "4                       No     Ch          Yes        <30  \n",
       "..                     ...    ...          ...        ...  \n",
       "595                     No     Ch          Yes        >30  \n",
       "596                     No     No          Yes        >30  \n",
       "597                     No     Ch          Yes         NO  \n",
       "598                     No     No          Yes         NO  \n",
       "599                     No     No           No         NO  \n",
       "\n",
       "[600 rows x 47 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No output_directory specified. Models will be saved in: AutogluonModels/ag-20200801_195938/\n",
      "Beginning AutoGluon training ... Time limit = 30s\n",
      "AutoGluon will save models to AutogluonModels/ag-20200801_195938/\n",
      "AutoGluon Version:  0.0.13b20200731\n",
      "Train Data Rows:    600\n",
      "Train Data Columns: 47\n",
      "Tuning Data Rows:    600\n",
      "Tuning Data Columns: 47\n",
      "Preprocessing data ...\n",
      "Here are the 3 unique label values in your data:  ['NO', '>30', '<30']\n",
      "AutoGluon infers your prediction problem is: multiclass  (because dtype of label-column == object).\n",
      "If this is wrong, please specify `problem_type` argument in fit() instead (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])\n",
      "\n",
      "Train Data Class Count: 3\n",
      "Feature Generator processed 1200 data points with 37 features\n",
      "Original Features (raw dtypes):\n",
      "\tobject features: 29\n",
      "\tfloat64 features: 1\n",
      "\tint64 features: 7\n",
      "Original Features (inferred dtypes):\n",
      "\tobject features: 29\n",
      "\tfloat features: 1\n",
      "\tint features: 7\n",
      "Generated Features (special dtypes):\n",
      "Processed Features (raw dtypes):\n",
      "\tfloat features: 1\n",
      "\tint features: 7\n",
      "\tcategory features: 29\n",
      "Processed Features:\n",
      "\tfloat features: 1\n",
      "\tint features: 7\n",
      "\tcategory features: 29\n",
      "\tData preprocessing and feature engineering runtime = 0.26s ...\n",
      "AutoGluon will gauge predictive performance using evaluation metric: accuracy\n",
      "To change this, specify the eval_metric argument of fit()\n",
      "AutoGluon will early stop models using evaluation metric: accuracy\n",
      "Fitting model: RandomForestClassifierGini ... Training model for up to 29.74s of the 29.74s of remaining time.\n",
      "\t0.4867\t = Validation accuracy score\n",
      "\t0.76s\t = Training runtime\n",
      "\t0.13s\t = Validation runtime\n",
      "Fitting model: RandomForestClassifierEntr ... Training model for up to 28.8s of the 28.8s of remaining time.\n",
      "\t0.47\t = Validation accuracy score\n",
      "\t0.77s\t = Training runtime\n",
      "\t0.13s\t = Validation runtime\n",
      "Fitting model: ExtraTreesClassifierGini ... Training model for up to 27.87s of the 27.87s of remaining time.\n",
      "\t0.4767\t = Validation accuracy score\n",
      "\t0.84s\t = Training runtime\n",
      "\t0.15s\t = Validation runtime\n",
      "Fitting model: ExtraTreesClassifierEntr ... Training model for up to 26.8s of the 26.8s of remaining time.\n",
      "\t0.475\t = Validation accuracy score\n",
      "\t0.59s\t = Training runtime\n",
      "\t0.14s\t = Validation runtime\n",
      "Fitting model: KNeighborsClassifierUnif ... Training model for up to 26.0s of the 26.0s of remaining time.\n",
      "\t0.43\t = Validation accuracy score\n",
      "\t0.0s\t = Training runtime\n",
      "\t0.11s\t = Validation runtime\n",
      "Fitting model: KNeighborsClassifierDist ... Training model for up to 25.87s of the 25.87s of remaining time.\n",
      "\t0.445\t = Validation accuracy score\n",
      "\t0.0s\t = Training runtime\n",
      "\t0.11s\t = Validation runtime\n",
      "Fitting model: LightGBMClassifier ... Training model for up to 25.75s of the 25.75s of remaining time.\n",
      "\t0.5333\t = Validation accuracy score\n",
      "\t1.59s\t = Training runtime\n",
      "\t0.04s\t = Validation runtime\n",
      "Fitting model: CatboostClassifier ... Training model for up to 24.1s of the 24.1s of remaining time.\n",
      "\t0.5433\t = Validation accuracy score\n",
      "\t6.9s\t = Training runtime\n",
      "\t0.03s\t = Validation runtime\n",
      "Fitting model: NeuralNetClassifier ... Training model for up to 17.15s of the 17.15s of remaining time.\n",
      "\t0.5583\t = Validation accuracy score\n",
      "\t11.52s\t = Training runtime\n",
      "\t0.15s\t = Validation runtime\n",
      "Fitting model: LightGBMClassifierCustom ... Training model for up to 5.45s of the 5.45s of remaining time.\n",
      "\t0.5333\t = Validation accuracy score\n",
      "\t1.39s\t = Training runtime\n",
      "\t0.03s\t = Validation runtime\n",
      "Fitting model: weighted_ensemble_k0_l1 ... Training model for up to 29.74s of the 2.66s of remaining time.\n",
      "\t0.56\t = Validation accuracy score\n",
      "\t0.52s\t = Training runtime\n",
      "\t0.0s\t = Validation runtime\n",
      "AutoGluon training complete, total runtime = 27.9s ...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Performance on your provided val_data:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>score_val</th>\n",
       "      <th>pred_time_val</th>\n",
       "      <th>fit_time</th>\n",
       "      <th>pred_time_val_marginal</th>\n",
       "      <th>fit_time_marginal</th>\n",
       "      <th>stack_level</th>\n",
       "      <th>can_infer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>weighted_ensemble_k0_l1</td>\n",
       "      <td>0.560000</td>\n",
       "      <td>0.177646</td>\n",
       "      <td>18.944229</td>\n",
       "      <td>0.001663</td>\n",
       "      <td>0.519197</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NeuralNetClassifier</td>\n",
       "      <td>0.558333</td>\n",
       "      <td>0.147606</td>\n",
       "      <td>11.522898</td>\n",
       "      <td>0.147606</td>\n",
       "      <td>11.522898</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>CatboostClassifier</td>\n",
       "      <td>0.543333</td>\n",
       "      <td>0.028377</td>\n",
       "      <td>6.902134</td>\n",
       "      <td>0.028377</td>\n",
       "      <td>6.902134</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>LightGBMClassifierCustom</td>\n",
       "      <td>0.533333</td>\n",
       "      <td>0.033193</td>\n",
       "      <td>1.385700</td>\n",
       "      <td>0.033193</td>\n",
       "      <td>1.385700</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>LightGBMClassifier</td>\n",
       "      <td>0.533333</td>\n",
       "      <td>0.035097</td>\n",
       "      <td>1.593748</td>\n",
       "      <td>0.035097</td>\n",
       "      <td>1.593748</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>RandomForestClassifierGini</td>\n",
       "      <td>0.486667</td>\n",
       "      <td>0.127345</td>\n",
       "      <td>0.762700</td>\n",
       "      <td>0.127345</td>\n",
       "      <td>0.762700</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>ExtraTreesClassifierGini</td>\n",
       "      <td>0.476667</td>\n",
       "      <td>0.151468</td>\n",
       "      <td>0.835468</td>\n",
       "      <td>0.151468</td>\n",
       "      <td>0.835468</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>ExtraTreesClassifierEntr</td>\n",
       "      <td>0.475000</td>\n",
       "      <td>0.135921</td>\n",
       "      <td>0.586289</td>\n",
       "      <td>0.135921</td>\n",
       "      <td>0.586289</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>RandomForestClassifierEntr</td>\n",
       "      <td>0.470000</td>\n",
       "      <td>0.126998</td>\n",
       "      <td>0.766957</td>\n",
       "      <td>0.126998</td>\n",
       "      <td>0.766957</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>KNeighborsClassifierDist</td>\n",
       "      <td>0.445000</td>\n",
       "      <td>0.111001</td>\n",
       "      <td>0.004122</td>\n",
       "      <td>0.111001</td>\n",
       "      <td>0.004122</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>KNeighborsClassifierUnif</td>\n",
       "      <td>0.430000</td>\n",
       "      <td>0.113201</td>\n",
       "      <td>0.003713</td>\n",
       "      <td>0.113201</td>\n",
       "      <td>0.003713</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         model  score_val  pred_time_val   fit_time  \\\n",
       "0      weighted_ensemble_k0_l1   0.560000       0.177646  18.944229   \n",
       "1          NeuralNetClassifier   0.558333       0.147606  11.522898   \n",
       "2           CatboostClassifier   0.543333       0.028377   6.902134   \n",
       "3     LightGBMClassifierCustom   0.533333       0.033193   1.385700   \n",
       "4           LightGBMClassifier   0.533333       0.035097   1.593748   \n",
       "5   RandomForestClassifierGini   0.486667       0.127345   0.762700   \n",
       "6     ExtraTreesClassifierGini   0.476667       0.151468   0.835468   \n",
       "7     ExtraTreesClassifierEntr   0.475000       0.135921   0.586289   \n",
       "8   RandomForestClassifierEntr   0.470000       0.126998   0.766957   \n",
       "9     KNeighborsClassifierDist   0.445000       0.111001   0.004122   \n",
       "10    KNeighborsClassifierUnif   0.430000       0.113201   0.003713   \n",
       "\n",
       "    pred_time_val_marginal  fit_time_marginal  stack_level  can_infer  \n",
       "0                 0.001663           0.519197            1       True  \n",
       "1                 0.147606          11.522898            0       True  \n",
       "2                 0.028377           6.902134            0       True  \n",
       "3                 0.033193           1.385700            0       True  \n",
       "4                 0.035097           1.593748            0       True  \n",
       "5                 0.127345           0.762700            0       True  \n",
       "6                 0.151468           0.835468            0       True  \n",
       "7                 0.135921           0.586289            0       True  \n",
       "8                 0.126998           0.766957            0       True  \n",
       "9                 0.111001           0.004122            0       True  \n",
       "10                0.113201           0.003713            0       True  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "from collections import Counter\n",
    "import pprint, psutil\n",
    "from autogluon import TabularPrediction as task\n",
    "\n",
    "subsample_size = 600 # experiment with larger values to try AutoGluon with larger datasets \n",
    "\n",
    "train_data = task.Dataset(file_path='https://autogluon.s3.amazonaws.com/datasets/diabetes/train.csv')\n",
    "val_data = task.Dataset(file_path='https://autogluon.s3.amazonaws.com/datasets/diabetes/validation.csv')\n",
    "train_data = train_data.head(subsample_size) # subsample data for faster demo\n",
    "val_data = val_data.head(subsample_size) # subsample data for faster demo\n",
    "display(train_data)\n",
    "display(val_data)\n",
    "\n",
    "label_column = 'readmitted'\n",
    "predictor = task.fit(train_data=train_data, tuning_data=val_data, label=label_column, time_limits=30)\n",
    "val_perf = predictor.leaderboard(silent=True)\n",
    "print(\"Performance on your provided val_data:\")\n",
    "display(val_perf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Refit on full dataset:** Regardless of how the validation dataset is selected, one way to often boost performance for models that train stably is to simply refit them to the entire dataset (training + validation) after their optimal hyperparameters (and training-iterations) have been determined based on the validation data.  However, the only way to confirm the resulting accuracy boost is through access to labeled test data, as there no longer remain any held-out examples in the dataset that can be utilized for unbiased accuracy estimation. With AutoGluon, one can do this as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: RandomForestClassifierGini_FULL ...\n",
      "\t0.86s\t = Training runtime\n",
      "Fitting model: RandomForestClassifierEntr_FULL ...\n",
      "\t0.85s\t = Training runtime\n",
      "Fitting model: ExtraTreesClassifierGini_FULL ...\n",
      "\t0.76s\t = Training runtime\n",
      "Fitting model: ExtraTreesClassifierEntr_FULL ...\n",
      "\t0.75s\t = Training runtime\n",
      "Fitting model: KNeighborsClassifierUnif_FULL ...\n",
      "\t0.0s\t = Training runtime\n",
      "Fitting model: KNeighborsClassifierDist_FULL ...\n",
      "\t0.0s\t = Training runtime\n",
      "Fitting model: LightGBMClassifier_FULL ...\n",
      "\t0.13s\t = Training runtime\n",
      "Fitting model: CatboostClassifier_FULL ...\n",
      "\t0.54s\t = Training runtime\n",
      "Fitting model: NeuralNetClassifier_FULL ...\n",
      "\t3.53s\t = Training runtime\n",
      "Fitting model: LightGBMClassifierCustom_FULL ...\n",
      "\t0.15s\t = Training runtime\n",
      "Fitting model: weighted_ensemble_FULL_k0_l1 ...\n",
      "\t0.56\t = Validation accuracy score\n",
      "\t0.03s\t = Training runtime\n",
      "\t0.0s\t = Validation runtime\n",
      "Loaded data from: https://autogluon.s3.amazonaws.com/datasets/diabetes/test.csv | Columns = 47 / 47 | Rows = 20354 -> 20354\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>score_test</th>\n",
       "      <th>score_val</th>\n",
       "      <th>pred_time_test</th>\n",
       "      <th>pred_time_val</th>\n",
       "      <th>fit_time</th>\n",
       "      <th>pred_time_test_marginal</th>\n",
       "      <th>pred_time_val_marginal</th>\n",
       "      <th>fit_time_marginal</th>\n",
       "      <th>stack_level</th>\n",
       "      <th>can_infer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>CatboostClassifier_FULL</td>\n",
       "      <td>0.601667</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.027697</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.543249</td>\n",
       "      <td>0.027697</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.543249</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>RandomForestClassifierGini_FULL</td>\n",
       "      <td>0.575000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.158222</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.861494</td>\n",
       "      <td>0.158222</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.861494</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>LightGBMClassifier_FULL</td>\n",
       "      <td>0.573333</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.031525</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.125560</td>\n",
       "      <td>0.031525</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.125560</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>LightGBMClassifierCustom</td>\n",
       "      <td>0.573333</td>\n",
       "      <td>0.533333</td>\n",
       "      <td>0.031607</td>\n",
       "      <td>0.033193</td>\n",
       "      <td>1.385700</td>\n",
       "      <td>0.031607</td>\n",
       "      <td>0.033193</td>\n",
       "      <td>1.385700</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>LightGBMClassifierCustom_FULL</td>\n",
       "      <td>0.573333</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.032832</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.149797</td>\n",
       "      <td>0.032832</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.149797</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>LightGBMClassifier</td>\n",
       "      <td>0.573333</td>\n",
       "      <td>0.533333</td>\n",
       "      <td>0.033500</td>\n",
       "      <td>0.035097</td>\n",
       "      <td>1.593748</td>\n",
       "      <td>0.033500</td>\n",
       "      <td>0.035097</td>\n",
       "      <td>1.593748</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>RandomForestClassifierEntr_FULL</td>\n",
       "      <td>0.571667</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.153388</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.848461</td>\n",
       "      <td>0.153388</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.848461</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>NeuralNetClassifier_FULL</td>\n",
       "      <td>0.570000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.149691</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.525632</td>\n",
       "      <td>0.149691</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.525632</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>weighted_ensemble_FULL_k0_l1</td>\n",
       "      <td>0.570000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.186244</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.098808</td>\n",
       "      <td>0.008856</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.029927</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>ExtraTreesClassifierGini_FULL</td>\n",
       "      <td>0.556667</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.195693</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.759400</td>\n",
       "      <td>0.195693</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.759400</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>NeuralNetClassifier</td>\n",
       "      <td>0.553333</td>\n",
       "      <td>0.558333</td>\n",
       "      <td>0.199066</td>\n",
       "      <td>0.147606</td>\n",
       "      <td>11.522898</td>\n",
       "      <td>0.199066</td>\n",
       "      <td>0.147606</td>\n",
       "      <td>11.522898</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>CatboostClassifier</td>\n",
       "      <td>0.550000</td>\n",
       "      <td>0.543333</td>\n",
       "      <td>0.022781</td>\n",
       "      <td>0.028377</td>\n",
       "      <td>6.902134</td>\n",
       "      <td>0.022781</td>\n",
       "      <td>0.028377</td>\n",
       "      <td>6.902134</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>ExtraTreesClassifierEntr_FULL</td>\n",
       "      <td>0.548333</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.196285</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.754333</td>\n",
       "      <td>0.196285</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.754333</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>weighted_ensemble_k0_l1</td>\n",
       "      <td>0.546667</td>\n",
       "      <td>0.560000</td>\n",
       "      <td>0.224519</td>\n",
       "      <td>0.177646</td>\n",
       "      <td>18.944229</td>\n",
       "      <td>0.002673</td>\n",
       "      <td>0.001663</td>\n",
       "      <td>0.519197</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>ExtraTreesClassifierEntr</td>\n",
       "      <td>0.540000</td>\n",
       "      <td>0.475000</td>\n",
       "      <td>0.155211</td>\n",
       "      <td>0.135921</td>\n",
       "      <td>0.586289</td>\n",
       "      <td>0.155211</td>\n",
       "      <td>0.135921</td>\n",
       "      <td>0.586289</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>RandomForestClassifierGini</td>\n",
       "      <td>0.536667</td>\n",
       "      <td>0.486667</td>\n",
       "      <td>0.142522</td>\n",
       "      <td>0.127345</td>\n",
       "      <td>0.762700</td>\n",
       "      <td>0.142522</td>\n",
       "      <td>0.127345</td>\n",
       "      <td>0.762700</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>ExtraTreesClassifierGini</td>\n",
       "      <td>0.525000</td>\n",
       "      <td>0.476667</td>\n",
       "      <td>0.174044</td>\n",
       "      <td>0.151468</td>\n",
       "      <td>0.835468</td>\n",
       "      <td>0.174044</td>\n",
       "      <td>0.151468</td>\n",
       "      <td>0.835468</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>RandomForestClassifierEntr</td>\n",
       "      <td>0.516667</td>\n",
       "      <td>0.470000</td>\n",
       "      <td>0.134201</td>\n",
       "      <td>0.126998</td>\n",
       "      <td>0.766957</td>\n",
       "      <td>0.134201</td>\n",
       "      <td>0.126998</td>\n",
       "      <td>0.766957</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>KNeighborsClassifierDist_FULL</td>\n",
       "      <td>0.458333</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.106107</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.002650</td>\n",
       "      <td>0.106107</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.002650</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>KNeighborsClassifierUnif_FULL</td>\n",
       "      <td>0.440000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.106641</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.003172</td>\n",
       "      <td>0.106641</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.003172</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>KNeighborsClassifierUnif</td>\n",
       "      <td>0.435000</td>\n",
       "      <td>0.430000</td>\n",
       "      <td>0.105265</td>\n",
       "      <td>0.113201</td>\n",
       "      <td>0.003713</td>\n",
       "      <td>0.105265</td>\n",
       "      <td>0.113201</td>\n",
       "      <td>0.003713</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>KNeighborsClassifierDist</td>\n",
       "      <td>0.435000</td>\n",
       "      <td>0.445000</td>\n",
       "      <td>0.116527</td>\n",
       "      <td>0.111001</td>\n",
       "      <td>0.004122</td>\n",
       "      <td>0.116527</td>\n",
       "      <td>0.111001</td>\n",
       "      <td>0.004122</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              model  score_test  score_val  pred_time_test  \\\n",
       "0           CatboostClassifier_FULL    0.601667        NaN        0.027697   \n",
       "1   RandomForestClassifierGini_FULL    0.575000        NaN        0.158222   \n",
       "2           LightGBMClassifier_FULL    0.573333        NaN        0.031525   \n",
       "3          LightGBMClassifierCustom    0.573333   0.533333        0.031607   \n",
       "4     LightGBMClassifierCustom_FULL    0.573333        NaN        0.032832   \n",
       "5                LightGBMClassifier    0.573333   0.533333        0.033500   \n",
       "6   RandomForestClassifierEntr_FULL    0.571667        NaN        0.153388   \n",
       "7          NeuralNetClassifier_FULL    0.570000        NaN        0.149691   \n",
       "8      weighted_ensemble_FULL_k0_l1    0.570000        NaN        0.186244   \n",
       "9     ExtraTreesClassifierGini_FULL    0.556667        NaN        0.195693   \n",
       "10              NeuralNetClassifier    0.553333   0.558333        0.199066   \n",
       "11               CatboostClassifier    0.550000   0.543333        0.022781   \n",
       "12    ExtraTreesClassifierEntr_FULL    0.548333        NaN        0.196285   \n",
       "13          weighted_ensemble_k0_l1    0.546667   0.560000        0.224519   \n",
       "14         ExtraTreesClassifierEntr    0.540000   0.475000        0.155211   \n",
       "15       RandomForestClassifierGini    0.536667   0.486667        0.142522   \n",
       "16         ExtraTreesClassifierGini    0.525000   0.476667        0.174044   \n",
       "17       RandomForestClassifierEntr    0.516667   0.470000        0.134201   \n",
       "18    KNeighborsClassifierDist_FULL    0.458333        NaN        0.106107   \n",
       "19    KNeighborsClassifierUnif_FULL    0.440000        NaN        0.106641   \n",
       "20         KNeighborsClassifierUnif    0.435000   0.430000        0.105265   \n",
       "21         KNeighborsClassifierDist    0.435000   0.445000        0.116527   \n",
       "\n",
       "    pred_time_val   fit_time  pred_time_test_marginal  pred_time_val_marginal  \\\n",
       "0             NaN   0.543249                 0.027697                     NaN   \n",
       "1             NaN   0.861494                 0.158222                     NaN   \n",
       "2             NaN   0.125560                 0.031525                     NaN   \n",
       "3        0.033193   1.385700                 0.031607                0.033193   \n",
       "4             NaN   0.149797                 0.032832                     NaN   \n",
       "5        0.035097   1.593748                 0.033500                0.035097   \n",
       "6             NaN   0.848461                 0.153388                     NaN   \n",
       "7             NaN   3.525632                 0.149691                     NaN   \n",
       "8             NaN   4.098808                 0.008856                     NaN   \n",
       "9             NaN   0.759400                 0.195693                     NaN   \n",
       "10       0.147606  11.522898                 0.199066                0.147606   \n",
       "11       0.028377   6.902134                 0.022781                0.028377   \n",
       "12            NaN   0.754333                 0.196285                     NaN   \n",
       "13       0.177646  18.944229                 0.002673                0.001663   \n",
       "14       0.135921   0.586289                 0.155211                0.135921   \n",
       "15       0.127345   0.762700                 0.142522                0.127345   \n",
       "16       0.151468   0.835468                 0.174044                0.151468   \n",
       "17       0.126998   0.766957                 0.134201                0.126998   \n",
       "18            NaN   0.002650                 0.106107                     NaN   \n",
       "19            NaN   0.003172                 0.106641                     NaN   \n",
       "20       0.113201   0.003713                 0.105265                0.113201   \n",
       "21       0.111001   0.004122                 0.116527                0.111001   \n",
       "\n",
       "    fit_time_marginal  stack_level  can_infer  \n",
       "0            0.543249            0       True  \n",
       "1            0.861494            0       True  \n",
       "2            0.125560            0       True  \n",
       "3            1.385700            0       True  \n",
       "4            0.149797            0       True  \n",
       "5            1.593748            0       True  \n",
       "6            0.848461            0       True  \n",
       "7            3.525632            0       True  \n",
       "8            0.029927            1       True  \n",
       "9            0.759400            0       True  \n",
       "10          11.522898            0       True  \n",
       "11           6.902134            0       True  \n",
       "12           0.754333            0       True  \n",
       "13           0.519197            1       True  \n",
       "14           0.586289            0       True  \n",
       "15           0.762700            0       True  \n",
       "16           0.835468            0       True  \n",
       "17           0.766957            0       True  \n",
       "18           0.002650            0       True  \n",
       "19           0.003172            0       True  \n",
       "20           0.003713            0       True  \n",
       "21           0.004122            0       True  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "refit_models = predictor.refit_full()\n",
    "test_data = task.Dataset(file_path='https://autogluon.s3.amazonaws.com/datasets/diabetes/test.csv')\n",
    "test_data = test_data.head(subsample_size) # subsample data for faster demo\n",
    "all_models_testperf = predictor.leaderboard(test_data, silent=True)\n",
    "display(all_models_testperf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above we list the test accuracy of all models/ensembles (including those originally fit to just `train_data` and those refit to the merged `train_data + val_data` indicated by suffix **_FULL**). The **_FULL** models lack validation scores as their performance cannot be reliably estimated without additional test-data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Preprocessing\n",
    "\n",
    "Properly processing raw data into a format suitable for ML is crucial for a successful end-to-end AutoML system.  \n",
    "AutoGluon relies on two sequential stages of data processing: \n",
    "- *model-agnostic* preprocessing that transforms the inputs to all models\n",
    "- *model-specific* preprocessing that is only applied to a copy of the data used to train a particular model. \n",
    "\n",
    "Model-agnostic preprocessing classifies each feature as numeric, categorical, text, or date/time, relying partly on the [**dtype** of each column in the DataFrame](https://pbpython.com/pandas_dtypes.html). Uncategorized columns are discarded from the data, comprised of non-numeric, non-repeating fields with presumably little\n",
    "predictive value (e.g. UserIDs). To deal with missing categorical variables, we create an\n",
    "additional **Unknown** category rather than imputing them.\n",
    "This strategy is also used by AutoGluon to handle previously unseen\n",
    "categories at inference-time. Note that often observations are not\n",
    "missing at random and we want to preserve the evidence of\n",
    "absence (rather than the absence of evidence). \n",
    "\n",
    "**Word of Caution about One-hot Encoding of Categorical Features**: \n",
    "Most ML textbooks/tutorials claim that categorical features should be converted to numerical values through [one-hot encoding](https://machinelearningmastery.com/why-one-hot-encode-data-in-machine-learning/) (OHE).  However, we do *not* recommend this as a generic model-agnostic preprocessing strategy; you should only apply OHE when passing data to a model for which this technique is particularly well-suited. OHE comes with the major downside that it explodes the dimsensionality of your data, and numerous alternative categorical-processing techniques exist ([Grover, 2019](https://towardsdatascience.com/getting-deeper-into-categorical-encodings-for-machine-learning-2312acd347c8)). \n",
    "Certain models such as [LightGBM](https://lightgbm.readthedocs.io/en/latest/Advanced-Topics.html#categorical-feature-support)/[Catboost](https://catboost.ai/docs/features/categorical-features.html) provide special handling of categoricals, as does AutoGluon's neural network, which utilizes [learned embeddings](https://www.fast.ai/2018/04/29/categorical-embeddings/) to represent categorical data ([Erickson, 2020](https://arxiv.org/abs/2003.06505)).  \n",
    "\n",
    "If you do utilize one-hot encoding, make sure to consider how your ML system will handle categorical features with: missing values, a huge number of possible categories, or previously-unseen categories encountered in future test data. One way to handle all of these issues is to provision separate OHE-dimensions to only the top $K$ most commonly-occurring categories and bin all less common (and previously-unseen/missing) categories into a single extra category."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Special Features**:\n",
    "We identify text features as columns of mostly unique strings, which on average contain more than 3 non-adjacent whitespace characters. For models that solely operate on numerical/categorical data, the values of each text column are encoded via numeric vectors of [n-gram features](https://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html) (only retaining those n-grams\n",
    "with high overall occurrence, 30+ times by default in the text columns to reduce\n",
    "memory footprint). In addition to n-grams, AutoGluon-Tabular also generates additional text features including: the number of whitespaces and the average word length in each text field.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"files/images/ngram.png\" width=\"900\" height=\"400\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Date/time features are also transformed\n",
    "into ordered numeric values via the following simple [pandas](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.to_datetime.html) transformation: `pd.to_numeric(pd.to_datetime(raw_feature))`. Richer date/time feature-engineering is provided in the [fast.ai library](https://docs.fast.ai/tabular.transform.html#add_datepart), but our naive approach works reasonably well in practice. \n",
    "After encoding text and date-times, a copy of the resulting set\n",
    "of numeric and categorical features is subsequently passed\n",
    "to model-specific methods for further tailored preprocessing specific to each model. Below we add a dummy 'text' column to our dataset to show how AutoGluon handles it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0      caaaccbcb  baaacc aaa aaa  aaabaaaaca cb\n",
       "1              baaabcaccc cbbcaccbabbcb abacbbb\n",
       "2              aacaab a a b caaaabababbaccbcac \n",
       "3            bcaaaabaca acbaaa bcccb b cb b c c\n",
       "4          aaac cbcabcbcbabcccccb bb  aaaa aaa \n",
       "                         ...                   \n",
       "595      b caaabaaaabaaaaaab b a ba b  ccb  acb\n",
       "596          ababb bc  bcaaabaaabccbcab b abcb \n",
       "597           baccbbcaaaaabca c  cba aaaa  baa \n",
       "598              c  b bab bccbcbb    ccaa aaaa \n",
       "599            cbc aa baccccacbbc babaaabcb cb \n",
       "Name: dummytext, Length: 600, dtype: object"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No output_directory specified. Models will be saved in: AutogluonModels/ag-20200801_200022/\n",
      "Beginning AutoGluon training ... Time limit = 30s\n",
      "AutoGluon will save models to AutogluonModels/ag-20200801_200022/\n",
      "AutoGluon Version:  0.0.13b20200731\n",
      "Train Data Rows:    600\n",
      "Train Data Columns: 48\n",
      "Preprocessing data ...\n",
      "Here are the 3 unique label values in your data:  ['NO', '>30', '<30']\n",
      "AutoGluon infers your prediction problem is: multiclass  (because dtype of label-column == object).\n",
      "If this is wrong, please specify `problem_type` argument in fit() instead (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])\n",
      "\n",
      "Train Data Class Count: 3\n",
      "Feature Generator processed 600 data points with 50 features\n",
      "Original Features (raw dtypes):\n",
      "\tobject features: 26\n",
      "\tfloat64 features: 1\n",
      "\tint64 features: 7\n",
      "Original Features (inferred dtypes):\n",
      "\tobject features: 25\n",
      "\tfloat features: 1\n",
      "\tint features: 7\n",
      "\ttext features: 1\n",
      "Generated Features (special dtypes):\n",
      "\ttext_as_category features: 1\n",
      "\ttext_special features: 4\n",
      "\ttext_ngram features: 12\n",
      "Processed Features (raw dtypes):\n",
      "\tfloat features: 1\n",
      "\tint features: 23\n",
      "\tcategory features: 26\n",
      "Processed Features:\n",
      "\ttext_as_category features: 1\n",
      "\ttext_special features: 4\n",
      "\ttext_ngram features: 12\n",
      "\tfloat features: 1\n",
      "\tint features: 7\n",
      "\tcategory features: 25\n",
      "\tData preprocessing and feature engineering runtime = 0.34s ...\n",
      "AutoGluon will gauge predictive performance using evaluation metric: accuracy\n",
      "To change this, specify the eval_metric argument of fit()\n",
      "AutoGluon will early stop models using evaluation metric: accuracy\n",
      "Fitting model: RandomForestClassifierGini ... Training model for up to 29.66s of the 29.66s of remaining time.\n",
      "\t0.575\t = Validation accuracy score\n",
      "\t0.74s\t = Training runtime\n",
      "\t0.14s\t = Validation runtime\n",
      "Fitting model: RandomForestClassifierEntr ... Training model for up to 28.74s of the 28.74s of remaining time.\n",
      "\t0.55\t = Validation accuracy score\n",
      "\t0.77s\t = Training runtime\n",
      "\t0.13s\t = Validation runtime\n",
      "Fitting model: ExtraTreesClassifierGini ... Training model for up to 27.81s of the 27.81s of remaining time.\n",
      "\t0.5333\t = Validation accuracy score\n",
      "\t0.56s\t = Training runtime\n",
      "\t0.13s\t = Validation runtime\n",
      "Fitting model: ExtraTreesClassifierEntr ... Training model for up to 27.03s of the 27.03s of remaining time.\n",
      "\t0.5667\t = Validation accuracy score\n",
      "\t0.66s\t = Training runtime\n",
      "\t0.14s\t = Validation runtime\n",
      "Fitting model: KNeighborsClassifierUnif ... Training model for up to 26.17s of the 26.17s of remaining time.\n",
      "\t0.5\t = Validation accuracy score\n",
      "\t0.0s\t = Training runtime\n",
      "\t0.11s\t = Validation runtime\n",
      "Fitting model: KNeighborsClassifierDist ... Training model for up to 26.05s of the 26.05s of remaining time.\n",
      "\t0.5333\t = Validation accuracy score\n",
      "\t0.01s\t = Training runtime\n",
      "\t0.11s\t = Validation runtime\n",
      "Fitting model: LightGBMClassifier ... Training model for up to 25.93s of the 25.93s of remaining time.\n",
      "\t0.5833\t = Validation accuracy score\n",
      "\t0.39s\t = Training runtime\n",
      "\t0.03s\t = Validation runtime\n",
      "Fitting model: CatboostClassifier ... Training model for up to 25.49s of the 25.49s of remaining time.\n",
      "\t0.625\t = Validation accuracy score\n",
      "\t15.13s\t = Training runtime\n",
      "\t0.04s\t = Validation runtime\n",
      "Fitting model: NeuralNetClassifier ... Training model for up to 10.29s of the 10.29s of remaining time.\n",
      "\tRan out of time, stopping training early.\n",
      "\t0.5333\t = Validation accuracy score\n",
      "\t10.42s\t = Training runtime\n",
      "\t0.06s\t = Validation runtime\n",
      "Fitting model: weighted_ensemble_k0_l1 ... Training model for up to 29.66s of the -3.54s of remaining time.\n",
      "\t0.625\t = Validation accuracy score\n",
      "\t0.6s\t = Training runtime\n",
      "\t0.0s\t = Validation runtime\n",
      "AutoGluon training complete, total runtime = 34.18s ...\n"
     ]
    }
   ],
   "source": [
    "text_feature = [''.join(np.random.choice(['a','b','c','aaa',' '],p=[0.2,0.3,0.2,0.1,0.2],size=30)) for i in range(len(train_data))]\n",
    "train_data_wtext = train_data.copy()\n",
    "train_data_wtext['dummytext'] = text_feature\n",
    "display(train_data_wtext['dummytext'])\n",
    "\n",
    "predictor = task.fit(train_data=train_data_wtext, label=label_column, time_limits=30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AutoGluon assigned the raw features to the following types:\n",
      "{'category': ['gender',\n",
      "              'age',\n",
      "              'admission_type_id',\n",
      "              'discharge_disposition_id',\n",
      "              'admission_source_id',\n",
      "              'medical_specialty',\n",
      "              'diag_1',\n",
      "              'diag_2',\n",
      "              'diag_3',\n",
      "              'max_glu_serum',\n",
      "              'A1Cresult',\n",
      "              'metformin',\n",
      "              'repaglinide',\n",
      "              'glimepiride',\n",
      "              'glipizide',\n",
      "              'glyburide',\n",
      "              'tolbutamide',\n",
      "              'pioglitazone',\n",
      "              'rosiglitazone',\n",
      "              'acarbose',\n",
      "              'troglitazone',\n",
      "              'tolazamide',\n",
      "              'insulin',\n",
      "              'change',\n",
      "              'diabetesMed',\n",
      "              'dummytext'],\n",
      " 'float': ['time_in_hospital'],\n",
      " 'int': ['num_lab_procedures',\n",
      "         'num_procedures',\n",
      "         'num_medications',\n",
      "         'number_outpatient',\n",
      "         'number_emergency',\n",
      "         'number_inpatient',\n",
      "         'number_diagnoses',\n",
      "         'dummytext.char_count',\n",
      "         'dummytext.word_count',\n",
      "         'dummytext.symbol_count. ',\n",
      "         'dummytext.symbol_ratio. ',\n",
      "         '__nlp__.aa',\n",
      "         '__nlp__.aaa',\n",
      "         '__nlp__.aaaa',\n",
      "         '__nlp__.ab',\n",
      "         '__nlp__.ac',\n",
      "         '__nlp__.ba',\n",
      "         '__nlp__.bb',\n",
      "         '__nlp__.bc',\n",
      "         '__nlp__.ca',\n",
      "         '__nlp__.cb',\n",
      "         '__nlp__.cc',\n",
      "         '__nlp__._total_']}\n",
      "\n",
      " AutoGluon generated the following 'special' features:\n",
      "{'text_as_category': ['dummytext'],\n",
      " 'text_ngram': ['__nlp__.aa',\n",
      "                '__nlp__.cc',\n",
      "                '__nlp__.ca',\n",
      "                '__nlp__.ba',\n",
      "                '__nlp__.bc',\n",
      "                '__nlp__.cb',\n",
      "                '__nlp__.ab',\n",
      "                '__nlp__.bb',\n",
      "                '__nlp__.ac',\n",
      "                '__nlp__.aaaa',\n",
      "                '__nlp__._total_',\n",
      "                '__nlp__.aaa'],\n",
      " 'text_special': ['dummytext.symbol_ratio. ',\n",
      "                  'dummytext.symbol_count. ',\n",
      "                  'dummytext.word_count',\n",
      "                  'dummytext.char_count']}\n",
      "\n",
      " After model-agnostic processing, the data passed to individual models looks like this:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time_in_hospital</th>\n",
       "      <th>num_lab_procedures</th>\n",
       "      <th>num_procedures</th>\n",
       "      <th>num_medications</th>\n",
       "      <th>number_outpatient</th>\n",
       "      <th>number_emergency</th>\n",
       "      <th>number_inpatient</th>\n",
       "      <th>number_diagnoses</th>\n",
       "      <th>gender</th>\n",
       "      <th>age</th>\n",
       "      <th>...</th>\n",
       "      <th>__nlp__.aaaa</th>\n",
       "      <th>__nlp__.ab</th>\n",
       "      <th>__nlp__.ac</th>\n",
       "      <th>__nlp__.ba</th>\n",
       "      <th>__nlp__.bb</th>\n",
       "      <th>__nlp__.bc</th>\n",
       "      <th>__nlp__.ca</th>\n",
       "      <th>__nlp__.cb</th>\n",
       "      <th>__nlp__.cc</th>\n",
       "      <th>__nlp__._total_</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>216</th>\n",
       "      <td>3.9</td>\n",
       "      <td>34</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>534</th>\n",
       "      <td>3.4</td>\n",
       "      <td>73</td>\n",
       "      <td>5</td>\n",
       "      <td>14</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>462</th>\n",
       "      <td>1.0</td>\n",
       "      <td>48</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>11.0</td>\n",
       "      <td>67</td>\n",
       "      <td>2</td>\n",
       "      <td>25</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>401</th>\n",
       "      <td>2.4</td>\n",
       "      <td>31</td>\n",
       "      <td>1</td>\n",
       "      <td>17</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>581</th>\n",
       "      <td>8.7</td>\n",
       "      <td>46</td>\n",
       "      <td>3</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>1.9</td>\n",
       "      <td>28</td>\n",
       "      <td>0</td>\n",
       "      <td>15</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>536</th>\n",
       "      <td>4.3</td>\n",
       "      <td>37</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>457</th>\n",
       "      <td>5.6</td>\n",
       "      <td>47</td>\n",
       "      <td>1</td>\n",
       "      <td>17</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>422</th>\n",
       "      <td>3.2</td>\n",
       "      <td>39</td>\n",
       "      <td>3</td>\n",
       "      <td>12</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>480 rows × 50 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     time_in_hospital  num_lab_procedures  num_procedures  num_medications  \\\n",
       "216               3.9                  34               0                4   \n",
       "534               3.4                  73               5               14   \n",
       "462               1.0                  48               0                6   \n",
       "49               11.0                  67               2               25   \n",
       "401               2.4                  31               1               17   \n",
       "..                ...                 ...             ...              ...   \n",
       "581               8.7                  46               3               20   \n",
       "40                1.9                  28               0               15   \n",
       "536               4.3                  37               1                5   \n",
       "457               5.6                  47               1               17   \n",
       "422               3.2                  39               3               12   \n",
       "\n",
       "     number_outpatient  number_emergency  number_inpatient  number_diagnoses  \\\n",
       "216                  0                 0                 0                 1   \n",
       "534                  0                 0                 0                 9   \n",
       "462                  0                 0                 2                 7   \n",
       "49                   0                 0                 0                 9   \n",
       "401                  0                 0                 0                 4   \n",
       "..                 ...               ...               ...               ...   \n",
       "581                  0                 0                 0                 8   \n",
       "40                   0                 0                 0                 4   \n",
       "536                  0                 0                 0                 7   \n",
       "457                  0                 0                 0                 7   \n",
       "422                  0                 0                 0                 5   \n",
       "\n",
       "    gender age  ... __nlp__.aaaa __nlp__.ab __nlp__.ac __nlp__.ba __nlp__.bb  \\\n",
       "216      0   0  ...            0          0          0          1          1   \n",
       "534      1   6  ...            1          0          0          0          0   \n",
       "462      1   1  ...            0          0          0          0          0   \n",
       "49       1   6  ...            0          0          0          0          0   \n",
       "401      1   8  ...            0          0          0          0          0   \n",
       "..     ...  ..  ...          ...        ...        ...        ...        ...   \n",
       "581      1   8  ...            0          0          0          0          0   \n",
       "40       0   7  ...            0          0          0          0          0   \n",
       "536      0   6  ...            0          0          0          0          0   \n",
       "457      1   7  ...            0          0          0          0          0   \n",
       "422      0   8  ...            0          0          0          0          0   \n",
       "\n",
       "    __nlp__.bc __nlp__.ca __nlp__.cb __nlp__.cc __nlp__._total_  \n",
       "216          0          0          0          0               2  \n",
       "534          0          0          0          0               1  \n",
       "462          0          0          0          0               0  \n",
       "49           0          0          0          0               0  \n",
       "401          0          0          0          0               0  \n",
       "..         ...        ...        ...        ...             ...  \n",
       "581          0          0          0          0               1  \n",
       "40           0          0          0          0               0  \n",
       "536          0          0          0          0               1  \n",
       "457          0          0          0          0               0  \n",
       "422          0          0          0          0               0  \n",
       "\n",
       "[480 rows x 50 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(\"AutoGluon assigned the raw features to the following types:\")\n",
    "pprint.pprint(dict(predictor.feature_types.feature_types_raw))\n",
    "\n",
    "print(\"\\n AutoGluon generated the following 'special' features:\")\n",
    "pprint.pprint(dict(predictor.feature_types.feature_types_special))\n",
    "\n",
    "print(\"\\n After model-agnostic processing, the data passed to individual models looks like this:\")\n",
    "processed_features, processed_labels = predictor.load_data_internal()\n",
    "display(processed_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Memory usage**: A key concern across AutoML is memory usage, as this often ends up being the bottleneck that causes AutoML systems to fail on certain larger datasets. \n",
    "Machines with sizeable RAM are now easily accessible in the cloud (eg. [AWS m5.24xlarge instance with 384 GB memory](https://aws.amazon.com/blogs/aws/m5-the-next-generation-of-general-purpose-ec2-instances/)), and thus robust AutoML systems ought to run on sizeable datasets even without any chunking/distributed-processing of the data. \n",
    "\n",
    "However, things break down without careful consideration of memory. For instance, many AutoML systems train multiple models in parallel which may lead to forking processes that duplicate datasets in memory. Similarly, when passing the generically-preprocessed data to a model, it is commmon to copy the dataset so that model-specific preprocessing won't affect the data other models receive. Without caution, additional copies of the dataset are often generated during this process or the model-training. AutoGluon's n-gram feature-generation is one particular area where memory becomes dangerous, as it often adds thousands of additional columns to the table. Here is how it is implemented:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "vectorizer = CountVectorizer(min_df=30, ngram_range=(1, 3), max_features=10000, dtype=np.uint8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use `uint8` to represent n-gram counts in the feature vector, which reduces memory. Before AutoGluon actually featurizes the text fields, we estimate how much memory the n-grams will require and downsample the number of n-grams iteratively until this estimate falls safely below the current available memory: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicted fraction of available memory used by n-grams: 0.07228432683881864\n"
     ]
    }
   ],
   "source": [
    "def get_ngram_freq(vectorizer, transform_matrix):\n",
    "    names = vectorizer.get_feature_names()\n",
    "    frequencies = transform_matrix.sum(axis=0).tolist()[0]\n",
    "    ngram_freq = {ngram: freq for ngram, freq in zip(names, frequencies)}\n",
    "    return ngram_freq\n",
    "\n",
    "def downscale_vectorizer(vectorizer, ngram_freq, vocab_size):\n",
    "    counter = Counter(ngram_freq)\n",
    "    top_n = counter.most_common(vocab_size)\n",
    "    top_n_names = sorted([name for name, _ in top_n])\n",
    "    new_vocab = {name: i for i, name in enumerate(top_n_names)}\n",
    "    vectorizer.vocabulary_ = new_vocab\n",
    "\n",
    "text_data = train_data_wtext['dummytext'].values\n",
    "vectorizer_fit = vectorizer.fit(text_data)\n",
    "transform_matrix = vectorizer_fit.transform(text_data)\n",
    "downsample_ratio = None\n",
    "predicted_ngrams_memory_usage_bytes = transform_matrix.shape[0] * 8 * (transform_matrix.shape[1] + 1) + 80\n",
    "mem_avail = psutil.virtual_memory().available\n",
    "mem_rss = psutil.Process().memory_info().rss\n",
    "max_memory_percentage = 0.15 # max fraction of available memory the n-grams are allowed to occupy\n",
    "predicted_rss = mem_rss + predicted_ngrams_memory_usage_bytes\n",
    "predicted_percentage = predicted_rss / mem_avail\n",
    "print(f\"Predicted fraction of available memory used by n-grams: {predicted_percentage}\")\n",
    "if downsample_ratio is None:\n",
    "    if predicted_percentage > max_memory_percentage:\n",
    "        downsample_ratio = max_memory_percentage / predicted_percentage\n",
    "\n",
    "if downsample_ratio is not None:\n",
    "    vocab_size = len(vectorizer_fit.vocabulary_)\n",
    "    downsampled_vocab_size = int(np.floor(vocab_size * downsample_ratio))\n",
    "    ngram_freq = get_ngram_freq(vectorizer=vectorizer_fit, transform_matrix=transform_matrix)\n",
    "    downscale_vectorizer(vectorizer=vectorizer_fit, ngram_freq=ngram_freq, vocab_size=downsampled_vocab_size)\n",
    "    transform_matrix = vectorizer_fit.transform(text_data)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Transductive Preprocessing**: While *supervised learning* typically assumes predictions will need to be made on future data that is currently unavailable, *transductive learning* [(Vapnik, 06)](http://axon.cs.byu.edu/~martinez/classes/778/Papers/transductive.pdf) instead only asks for predictions for one particular test dataset that is available (just without labels). For instance, many [prediction competitions](https://www.kaggle.com/competitions) follow this format, where the (unlabeled) test data is provided to contestants at the outset. While sophisticated learning algorithms have been developed specifically for the transductive setting, some accuracy-gains may be reaped simply by performing all data preprocessing on the combined training and (unlabeled) test datasets. While inappropriate for inductive supervised learning (where test data are merely supposed  provide unbiased predictive performance estimates for future data), such joint preprocessing can help in transductive settings (where in the test data: some numerical values may extend beyond their range in the training data and some categorical variables may take values previously unseen in the training data).\n",
    "This can be done in AutoGluon by passing the *unlabeled* test data as `tuning_data` (rather than *labeled* validation data):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No output_directory specified. Models will be saved in: AutogluonModels/ag-20200801_200056/\n",
      "Beginning AutoGluon training ... Time limit = 30s\n",
      "AutoGluon will save models to AutogluonModels/ag-20200801_200056/\n",
      "AutoGluon Version:  0.0.13b20200731\n",
      "Train Data Rows:    600\n",
      "Train Data Columns: 47\n",
      "Tuning Data Rows:    600\n",
      "Tuning Data Columns: 46\n",
      "Preprocessing data ...\n",
      "Here are the 3 unique label values in your data:  ['NO', '>30', '<30']\n",
      "AutoGluon infers your prediction problem is: multiclass  (because dtype of label-column == object).\n",
      "If this is wrong, please specify `problem_type` argument in fit() instead (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])\n",
      "\n",
      "Train Data Class Count: 3\n",
      "Feature Generator processed 1200 data points with 37 features\n",
      "Original Features (raw dtypes):\n",
      "\tobject features: 29\n",
      "\tfloat64 features: 1\n",
      "\tint64 features: 7\n",
      "Original Features (inferred dtypes):\n",
      "\tobject features: 29\n",
      "\tfloat features: 1\n",
      "\tint features: 7\n",
      "Generated Features (special dtypes):\n",
      "Processed Features (raw dtypes):\n",
      "\tfloat features: 1\n",
      "\tint features: 7\n",
      "\tcategory features: 29\n",
      "Processed Features:\n",
      "\tfloat features: 1\n",
      "\tint features: 7\n",
      "\tcategory features: 29\n",
      "\tData preprocessing and feature engineering runtime = 0.25s ...\n",
      "AutoGluon will gauge predictive performance using evaluation metric: accuracy\n",
      "To change this, specify the eval_metric argument of fit()\n",
      "AutoGluon will early stop models using evaluation metric: accuracy\n",
      "Fitting model: RandomForestClassifierGini ... Training model for up to 29.75s of the 29.75s of remaining time.\n",
      "\t0.6083\t = Validation accuracy score\n",
      "\t0.82s\t = Training runtime\n",
      "\t0.14s\t = Validation runtime\n",
      "Fitting model: RandomForestClassifierEntr ... Training model for up to 28.75s of the 28.75s of remaining time.\n",
      "\t0.625\t = Validation accuracy score\n",
      "\t0.83s\t = Training runtime\n",
      "\t0.14s\t = Validation runtime\n",
      "Fitting model: ExtraTreesClassifierGini ... Training model for up to 27.75s of the 27.75s of remaining time.\n",
      "\t0.5833\t = Validation accuracy score\n",
      "\t0.7s\t = Training runtime\n",
      "\t0.14s\t = Validation runtime\n",
      "Fitting model: ExtraTreesClassifierEntr ... Training model for up to 26.81s of the 26.81s of remaining time.\n",
      "\t0.5917\t = Validation accuracy score\n",
      "\t0.69s\t = Training runtime\n",
      "\t0.14s\t = Validation runtime\n",
      "Fitting model: KNeighborsClassifierUnif ... Training model for up to 25.91s of the 25.91s of remaining time.\n",
      "\t0.5\t = Validation accuracy score\n",
      "\t0.01s\t = Training runtime\n",
      "\t0.11s\t = Validation runtime\n",
      "Fitting model: KNeighborsClassifierDist ... Training model for up to 25.8s of the 25.8s of remaining time.\n",
      "\t0.5333\t = Validation accuracy score\n",
      "\t0.01s\t = Training runtime\n",
      "\t0.1s\t = Validation runtime\n",
      "Fitting model: LightGBMClassifier ... Training model for up to 25.67s of the 25.67s of remaining time.\n",
      "\t0.65\t = Validation accuracy score\n",
      "\t3.12s\t = Training runtime\n",
      "\t0.03s\t = Validation runtime\n",
      "Fitting model: CatboostClassifier ... Training model for up to 22.5s of the 22.5s of remaining time.\n",
      "\t0.6333\t = Validation accuracy score\n",
      "\t26.51s\t = Training runtime\n",
      "\t0.04s\t = Validation runtime\n",
      "Fitting model: weighted_ensemble_k0_l1 ... Training model for up to 29.75s of the -5.11s of remaining time.\n",
      "\t0.65\t = Validation accuracy score\n",
      "\t0.44s\t = Training runtime\n",
      "\t0.0s\t = Validation runtime\n",
      "AutoGluon training complete, total runtime = 35.6s ...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predictive performance on given dataset: accuracy = 0.53\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.53"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_data_nolab = test_data.drop(labels=[label_column],axis=1) # delete label column to demonstrate case without labels\n",
    "predictor = task.fit(train_data=train_data, tuning_data=test_data_nolab, label=label_column, time_limits=30)\n",
    "predictor.evaluate(test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "[**AutoGluon Documentation** (autogluon.mxnet.io)](https://autogluon.mxnet.io/api/autogluon.task.html)\n",
    "\n",
    "Rencberoglu, E. [**Fundamental Techniques of Feature Engineering for Machine Learning**](https://towardsdatascience.com/feature-engineering-for-machine-learning-3a5e293a5114). *Towards Data Science*, 2019.\n",
    "\n",
    "D'yakonov A, Semenov, S. [**Feature Preprocessing and Generation with Respect to Models**](https://www.coursera.org/lecture/competitive-data-science/overview-1Nh5Q). *From Coursera Course: [How to Win a Data Science Competition](https://sites.google.com/view/raybellwaves/courses/how-to-win-a-data-science-competition)*, 2019.\n",
    "\n",
    "Grover, P. [**Getting Deeper into Categorical Encodings for Machine Learning**](https://towardsdatascience.com/getting-deeper-into-categorical-encodings-for-machine-learning-2312acd347c8). *Towards Data Science*, 2019.\n",
    "\n",
    "Thomas, R. [**An Introduction to Deep Learning for Tabular Data**](https://www.fast.ai/2018/04/29/categorical-embeddings/). *fast.ai*, 2018.\n",
    "\n",
    "Erickson et al. [**AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data**](https://arxiv.org/abs/2003.06505). *Arxiv*, 2020.\n",
    "\n",
    "Vapnik V. [**Transductive Inference and Semi-Supervised Learning**](http://axon.cs.byu.edu/~martinez/classes/778/Papers/transductive.pdf). *Book Chapter in \"Semi-Supervised Learning\"*, 2006 "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
